Bidirectional pairing architecture for object detection in video

ABSTRACT

Techniques related to training and implementing a bidirectional pairing architecture for object detection are discussed. Such techniques include generating a first enhanced feature map for each frame of a video sequence by processing the frames in a first direction, generating a second enhanced feature map for frame by processing the frames in a second direction opposite the first, and determining object detection information for each frame using the first and second enhanced feature map for the frame.

CLAIM OF PRIORITY

This Application is a continuation of U.S. patent application Ser. No.17/059,968, filed on Nov. 30, 2020 and titled “BIDIRECTIONAL PAIRINGARCHITECTURE FOR OBJECT IN VIDEO”, which is a National Stage Entry of,and claims priority to, PCT Application No. PCT/CN2018/121736 filed onDec. 18, 2018 and titled “BIDIRECTIONAL PAIRING ARCHITECTURE FOR OBJECTIN VIDEO”, both are incorporated by reference in its entirety for allpurposes.

BACKGROUND

Object detection, including offline object detection, in video has greatpractical value and application prospects. For example, object detectionis an important prerequisite for many high-level visual processing andanalysis tasks such as behavior analysis, criminal investigation, eventdetection, scene semantic understanding, video summary, videorecommendation, person re-identification, etc. In some contexts, it isimportant to detect each object as much as possible such that falsenegatives are to be avoided and false positives are tolerable. Forexample, in criminal investigation fields, false negative detection inframes, particularly key frames, causes failed event detection orfailure in semantic understanding of a scene, which ultimately increasescriminal investigation difficulty. Other contexts for object detectionalso rely on low false negative rate detection in each frame such asautomatic driving, robot vision, etc.

Current object detection techniques include still image detection andvideo detection. However, such techniques may fail to detect objects ina variety of situations such as when portions of the object are obscuredby other objects. It may be advantageous to perform object detectionwith high accuracy and very low false negative rates. It is with respectto these and other considerations that the present improvements havebeen needed. Such improvements may become critical as the desire toperform object detection in video becomes more widespread.

BRIEF DESCRIPTION OF THE DRAWINGS

The material described herein is illustrated by way of example and notby way of limitation in the accompanying figures. For simplicity andclarity of illustration, elements illustrated in the figures are notnecessarily drawn to scale. For example, the dimensions of some elementsmay be exaggerated relative to other elements for clarity. Further,where considered appropriate, reference labels have been repeated amongthe figures to indicate corresponding or analogous elements. In thefigures:

FIG. 1 illustrates an example device for performing object detection orrecognition in video using a bidirectional pairing architecture;

FIG. 2 illustrates example video frames having an object that ispartially obscured for detection using device;

FIG. 3 illustrates an example bidirectional pairing architectureinference network for performing object detection or recognition invideo;

FIG. 4 illustrates example bounding boxes or patches of an exampleframe;

FIG. 5 is a diagram of exemplary local searching pairing for adjacentframes to a current frame;

FIG. 6 illustrates an example bidirectional palling architecturetraining network for performing object detection or recognition invideo;

FIG. 7 is a flow chart illustrating a process for training abidirectional pairing architecture to perform object detection orrecognition in video neural network to detect objects in video frames;

FIG. 8 is a flow chart illustrating a process for performing objectdetection or recognition in video using a bidirectional pairingarchitecture;

FIG. 9 is a flow diagram illustrating an example process for performingobject detection;

FIG. 10 is an illustrative diagram of an example system for performingobject detection;

FIG. 11 is an illustrative diagram of an example system; and

FIG. 12 illustrates an example device, all arranged in accordance withat least some implementations of the present disclosure.

DETAILED DESCRIPTION

One or more embodiments or implementations are now described withreference to the enclosed figures. While specific configurations andarrangements are discussed, it should be understood that this is donefor illustrative purposes only. Persons skilled in the relevant art willrecognize that other configurations and arrangements may be employedwithout departing from the spirit and scope of the description. It willbe apparent to those skilled in the relevant art that techniques and/orarrangements described herein may also be employed in a variety of othersystems and applications other than what is described herein.

While the following description sets forth various implementations thatmay be manifested in architectures such as system-on-a-chip (SoC)architectures for example, implementation of the techniques and/orarrangements described herein are not restricted to particulararchitectures and/or computing systems and may be implemented by anyarchitecture and/or computing system for similar purposes. For instance,various architectures employing, for example, multiple integratedcircuit (IC) chips and/or packages, and/or various computing devicesand/or consumer electronic (CE) devices such as set top boxes, smartphones, etc., may implement the techniques and/or arrangements describedherein. Further, while the following description may set forth numerousspecific details such as logic implementations, types andinterrelationships of system components, logic partitioning/integrationchoices, etc., claimed subject matter may be practiced without suchspecific details. In other instances, some material such as, forexample, control structures and full software instruction sequences, maynot be shown in detail in order not to obscure the material disclosedherein.

The material disclosed herein may be implemented in hardware, firmware,software, or any combination thereof. The material disclosed herein mayalso be implemented as instructions stored on a machine-readable medium,which may be read and executed by one or more processors. Amachine-readable medium may include any medium and/or mechanism forstoring or transmitting information in a form readable by a machine(e.g., a computing device). For example, a machine-readable medium mayinclude read only memory (ROM); random access memory (RAM); magneticdisk storage media; optical storage media; flash memory devices;electrical, optical, acoustical or other forms of propagated signals(e.g., carrier waves, infrared signals, digital signals, etc.), andothers.

References in the specification to “one implementation”, “animplementation”, “an example implementation”, etc., indicate that theimplementation described may include a particular feature, structure, orcharacteristic, but every embodiment may not necessarily include theparticular feature, structure, or characteristic. Moreover, such phrasesare not necessarily referring to the same implementation. Further, whena particular feature, structure, or characteristic is described inconnection with an embodiment, it is submitted that it is within theknowledge of one skilled in the art to affect such feature, structure,or characteristic in connection with other implementations whether ornot explicitly described herein. As used herein the terms“approximately” or “substantially” indicate a deviation from the targetvalue of +1-5% unless otherwise specified.

Methods, devices, apparatuses, computing platforms, and articles aredescribed herein related to bidirectional object recognition in videousing still image object recognition on each frame and similaritydetection between frames for improved object recognition.

As described above, it may be advantageous to perform object detectionand recognition with very low false negative rates in a variety ofcontexts. As used herein, the terms object detection and objectrecognition are used interchangeably and indicate detecting orrecognizing instances of objects of a certain class (e.g., humans,buildings, animals, cars, etc.) in digital video. The techniquesdiscussed herein use bidirectional object recognition techniques forobject detection. In particular, the discussed techniques considersuccessor and predecessor video information at the same time and detectkey frames as much as possible to improve detection accuracy for video,such as offline video, to better support high-level visual processingtasks.

In some embodiments, still image object detection is performed on eachof a current frame, a previous frame, and a subsequent frame of video(e.g., on all frames of a video sequence) to determine initial featuremaps comprising object detection localization and confidence scoring foreach of multiple potential object patches for the frames. As usedherein, the term feature map indicates a tensor that contains predictionresults (e.g., feature values) for a frame. The term prediction resultindicates a detected class, a confidence score and, optionally,localization information. In some embodiments, a feature map may includemultiple feature maps. For example, a feature map for a frame mayinclude a confidence feature map (or confidence information) and alocalization feature map (or localization information). The confidencefeature map indicates what is detected (e.g., a class: person, building,car, background etc.) and the confidence in the detection (e.g., a scoreranging from 0 to 1). For example, a person confidence map only includesperson class and score combinations (e.g., “person”, 0.78), while a carconfidence map only includes car class and score combinations (e.g.,“car”, 0.91). For one-stage object detection (e.g., YOLO, SSD), thelocalization feature map indicates a location or location offset basedon a bounding box of the detection. For example, each of any number ofdefault or predefined bounding boxes may be defined and indexed. As usedherein, the terms bounding box, region, and patch are usedinterchangeably. The localization feature map may then indicate thelocation (by index or coordinates) offset based on default or predefinedbounding boxes of the features or information in the previouslydiscusses confidence feature map. Together, the confidence feature mapand localization feature map thereby provide a classification,confidence score, and localization offset based on some or all of thepredefined bounding boxes. For two-stage object detection (e.g., RFCN),the localization feature map indicates a location, not location offset,because there are no default or predefined bounding boxes for two-stageobject detection. For example, each such classification, confidencescore, and localization offset based on default or predefined boundingboxes or localization may be characterized as a prediction result.Unless otherwise indicated, the discussion herein focuses on one-stageobject detection.

After still image object detection, paired patches between the currentand previous frames and paired patches between the current andsubsequent frames, respectively, are detected. Thereby, forward andbackward similar patches are detected. As used in this context, forwardrefers to forward direction along the video frames, while backwardrefers to the reverse direction along the video frames. As used in thiscontext, forward and backward are not the same concepts for training aneural network. Notably, such processing is performed in a forwarddirection along the video frames and in a reverse (or backward)direction along the video frames. In some embodiments, the results offorward processing are cached for later use as backward processing isperformed (or vice versa). Using the detected paired patches, updatesmay be made to the feature maps generated using still image detection.For example, for a pair of similar patches between the current frame anda previous frame, for the feature map of the current frame, a predictionresult of the feature map may be updated if the confidence in the pairedprediction (as previously stored in an enhanced feature map) result ishigher. In some embodiments, if the preliminary feature map for thecurrent frame has a particular patch as “dog”, 0.5 and the paired patchfrom the previous frame is scored as “cat”, 0.8, the prediction resultfor the preliminary feature map for the current frame is updated to“cat”, 0.8 for that patch. As discussed, with respect to the currentframe, such processing is performed in a forward direction (thusaccumulating updates from all previous frames) as processing continuesin a forward manner through all video frames. Then, such processing isperformed in a reverse direction (thus accumulating updates from allsubsequent frames) as processing continues in a reverse or backwardmanner through all video frames. As used herein, the terms forward,backward, etc. are meant to indicate bidirectional processing of videoonce along the temporal direction of video and once along the directionopposite the temporal direction. Such forward processing may beperformed first and reverse processing may be performed second or viceversa throughout.

As discussed, the preliminary feature map for the current frame isupdated through forward processing to generate a forward resultant orenhanced feature map. The preliminary feature map for the current frameis also updated through reverse processing to generate a reverse orenhanced resultant feature map. As noted above, both the forward andreverse feature maps may each include a confidence feature map and alocalization feature map. Using the forward and reverse feature maps,object detection results for the current frame are generated. In someembodiments, the object detection results are generated by concatenatingthe forward and reverse feature maps and providing them to a pretrainedneural network layer or layers (optionally including a softmaxfunction). In other embodiments, the object detection results aregenerated by comparing shared object patches and retaining theprediction result having the higher confidence. For example, if aparticular bounding box or patch has a score of “cat”, 0.6 in theforward feature map and a result of “dog”, 0.8 in the reverse featuremap (e.g., the prediction results are for the same patch index), theprediction result of “dog”, 0.8 is retained as an object detectionresult for the current frame. Thereby, a bidirectional enhanced featuremap is generated for the frame. The output for the current frame maythen include prediction results having confidence values exceeding aparticular threshold, a top number of prediction results (prioritized byconfidence values), or the like. In some embodiments, the bidirectionalenhanced feature map may be provided to one or more trained neuralnetwork layers and/or a softmax function to determine final objectdetection information.

In some embodiments, a frame sequence is reversed and object detectionin one direction of the video clip is performed and the results arecached (this utilizes the successor video information). Then objectdetection is performed in the opposite direction for the same video clip(this utilizes the predecessor video information). The cached featuremap and the current feature map are concatenated and provided to anelection layer, which determines the final detection output. In someembodiments, the election layer is a pretrained layer (e.g., includesone or more neural network layers). In some embodiments, the electionlayer is predetermined to retain higher confidence scores as discussed.

FIG. 1 illustrates an example device 100 for performing object detectionor recognition in video using a bidirectional pairing architecture,arranged in accordance with at least some implementations of the presentdisclosure. As shown in FIG. 1 , device 100 includes any number of stillimage object detectors 101, 102, 103, 104, any number of similaritydetectors 105, 106, 107, any number of concatenation modules 131, 132,133, 134, and any number of election modules 151, 152, 153, 154.Notably, device 100 may implement any number of such detectors andmodules such as one each, two each, etc. and such available detectorsand modules may be used in a sequential manner to perform the processingdiscussed herein. Device 100 may be implemented in any suitable formfactor device such as a personal computer, a laptop computer, a tablet,a motor vehicle platform, a robotics platform, a phablet, a smart phone,a digital camera, a gaming console, a wearable device, a display device,an all-in-one device, a two-in-one device, or the like. For example,device 100 may perform object detection as discussed herein. Forexample, device 100 performs forward object detection to generateforward enhanced feature maps, reverse object detection to generatereverse enhanced feature maps, and final detection using the forward andreverse enhanced feature maps. For example, device 100 may include amemory to store frames of video and a processor to perform theoperations discussed herein.

As shown, a sequence of video frames 111, 112, 113, 114 are provided inthe following temporal order: F1, F2, F3, . . . FN. Video frames 111,112, 113, 114 may include any suitable video frames, video pictures,sequence of video frames, group of pictures, groups of pictures, videodata, or the like in any suitable resolution. For example, the video maybe video graphics array (VGA), high definition (HD), Full-HD (e.g.,1080p), 4K resolution video, 5K resolution video, or the like, and thevideo may include any number of video frames, sequences of video frames,pictures, groups of pictures, or the like. Techniques discussed hereinare discussed with respect to frames for the sake of clarity ofpresentation. However, such frames may be characterized as videopictures.

FIG. 2 illustrates example video frames having an object that ispartially obscured for detection using device 100, arranged inaccordance with at least some implementations of the present disclosure.As shown in FIG. 2 , frames 201, 202, 203, 204 (also labeled as frame1,frame2, frame 3, frame4) include an object 210 (e.g., an animal) thatgoes in and out of being obscured by trees of frames 201, 202, 203, 204.Notably, using still image object detection, some of frames 201, 202,203, 204 may fail in detecting or recognizing object 210 due to it beingobscured. Furthermore, processing in the forward direction may detectobject 210 in frame 203 and frame 204, but not in frame 201, 202. Or,such forward processing may not detect object 210 at all. The techniquesdiscussed herein use both forward processing (e.g., frames in the orderframes 201, 202, 203, 204) and reverse processing (e.g., frames in theorder frames 204, 203, 202, 201) to detect object 210 in all frames suchthat all of frames 201, 202, 203, 204 may be detected as key framesframes having the object) for further processing.

Returning to FIG. 1 , as shown, in forward processing 171, frame 111 isprocessed by still object detector 104 and frame 112 is processed bystill object detector 103. Each of still object detector 104 and stillobject detector 103 generates an initial feature map based on theirrespective frames. As discussed, each of the feature maps may include aconfidence feature map (e.g., indicating what, if anything, has beendetected at each location) and a localization feature map indicatingwhere the feature is located. For example, each of the feature maps mayinclude a prediction result for each of thousands of potential objectpatches within each frame. Such feature map characteristics arediscussed further herein with respect to FIG. 4 . Also as shown,similarity detector 107 is applied for frames 111, 112 to detect similarpatches between frame 111 and frame 112. For similar patches, thepreliminary feature map of frame 112 is updated as discussed to generatea forward feature map 126. For example, for corresponding similarpatches, the prediction result for frame 112 is updated when theconfidence for the detected object in the patch of frame 111 exceeds theconfidence for the detected object in the patch of frame 112.

With reference to FIG. 2 , an animal is detected in a patch 211 of frame203 and a similar patch 212 is detected in frame 204 (notably the imageswithin patches 211, 212 differ but have substantial similarities, whichmay exceed a similarity threshold). In the illustrated example, patch211 may have a prediction result of “cattle”, 0.8 while the predictionresult of patch 212 may be “dog”, 0.5 or even “background”, 0.6. In suchexamples, the prediction result for patch 212 in the preliminary featuremap for frame 204 is updated to “cattle”, 0.8 based on the confidence ofthe prediction result of similar patch 211 in frame 203 exceeding theconfidence of the prediction result of similar patch 212 (e.g., 0.8>0.6so use detection result “cattle”). In some embodiments, an object of anyconfidence replaces a result of “background”). In the illustrativeembodiment, a prediction result of “cattle”, 0.8 for patch 211 wouldreplace any confidence level of “background”.

Returning again to FIG. 1 , forward feature map 126 for frame 112 may becached and saved for later processing. Notably, no detection results maybe generated using only forward feature map 126. Furthermore, a forwardfeature map 128 may be generated for frame 111 although no enhancementsusing another feature map may be generated (as frame 111 is the first inthe forward sequence). Such processing is repeated in a sequentialmanner for forward processing 171. Enhanced forward feature map 126 isused in the same manner to enhance a preliminary feature map for frame113 generated by still image object detector 102 based on similarpatches detected by similarity detector 106. The resultant enhancedforward feature map is shown as forward feature map 124, which again maybe cached for future use. Similar processing is repeated for any numberof frames through frame 114 (by still image object detectors includingstill image object detector 101 and similarity detectors includingsimilarity detector 105) to generate forward feature map 122 (and anynumber of intervening feature maps), all of which may be cached forlater use. Although discussed with respect to forward processing 171occurring prior to reverse processing 172, the forward and reverseprocessing may be performed in either order.

In reverse processing 172, frame 114 is processed by still objectdetector 101 and frame 113 is processed by still object detector 102. Asdiscussed with respect to forward processing 171, each of still objectdetector 101 and still object detector 102 generates a feature map basedon their respective frames. Notably, such still image object detectionmay only be performed once for each frame with the results being cached(still image object detection does not need to be repeated in forwardand reverse processing). Each of the feature maps, as described above,may include a confidence feature map and a localization feature map.Similarity detector 105 is applied to frames 114, 113 to detect similarpatches between frame 114 and frame 113 (again such processing may onlybe performed once with the results being cached). For similar patches,the preliminary feature map of frame 113 is updated to generate areverse feature map 123. Notably, each of the forward and reversefeature maps may be characterized as enhanced feature maps. Asdiscussed, for corresponding similar patches, the prediction result forframe 113 is updated when the confidence for the detected object in thepatch of frame 114 exceeds the confidence for the detected object in thepatch of frame 113.

Again with reference to FIG. 2 , an animal is detected in a patch 211 offrame 203 and a similar patch 213 is detected in frame 202 (notably theimages within patches 211, 213 differ but may have a measured similaritythat exceeds a similarity threshold). As discussed above, patch 211 mayhave a prediction result of “cattle”, 0.8. Notably, the predictionresult of patch 211 may have been from still image object detection orit may have been carried over from subsequent frames (e.g., framessubsequent to frame 204) due to the similarity and patch predictionresult processing discussed herein. Furthermore, the prediction resultof patch 213 may again be “dog”, 0.4 or even “background”, 0.8. In suchexamples, the prediction result for patch 213 in the preliminary featuremap for frame 202 is updated to “cattle”, 0.8 based on the confidence ofthe prediction result of similar patch 211 in frame 203 exceeding theconfidence of the prediction result of similar patch 213 (e.g., 0.8)-0.6so use detection result “cattle”). Similarly, the prediction result ofpatch 214 may again be updated to “cattle”, 0.8 based on the confidenceof the prediction result of similar patch 213 in frame 202 exceeding theconfidence of the prediction result of similar patch 214. Thereby, ahigh confidence detection in a subsequent frame is carried over toprevious frames (in the reverse processing iteration) to similarpatches. Such high confidence detection is also carried forward in thepreviously discussed forward processing iteration.

With reference to FIG. 1 , reverse feature map 123 and forward featuremap 124 (as attained from cache) for frame 113 are concatenated byconcatenation module 132 to generate concatenated feature maps 142. Asused herein, the term in concatenate indicates the reverse feature map123 and forward feature map 124 are joined without loss of information.In an embodiment, concatenation module 132 concatenates a confidencefeature map of reverse feature map 123 and a confidence feature map offorward feature map 124 separately from concatenation of a localizationfeature map of reverse feature map 123 and a localization feature map offorward feature map 124 to generate two concatenated feature maps: aconcatenated confidence feature map and a concatenated localizationfeature map. In either case, the resultant concatenated feature maps areillustrated as concatenated feature maps 142 corresponding to frame 113.Similarly, reverse feature map 121 and forward feature map 122 (asattained from cache) for frame 114 are concatenated by concatenationmodule 131 to generate concatenated feature maps 141, reverse featuremap 125 and forward feature map 126 (as attained from cache) for frame112 are concatenated by concatenation module 133 to generateconcatenated feature maps 143, reverse feature map 127 and forwardfeature map 128 (as attained from cache) for frame 111 are concatenatedby concatenation module 134 to generate concatenated feature maps 144,and so on to generate concatenated feature maps for any frames of thevideo sequence.

As shown, each of concatenated feature maps 141, 142, 143, 144 areprovided, respectively, to election modules 151, 152, 153, 154. Each ofelection modules 151, 152, 153, 154 may perform the same processing butany number of such election modules 151, 152, 153, 154 may beimplemented by device 100 for parallel processing. For each ofconcatenated feature maps 141, 142, 143, 144, resultant object detectiondata 161, 162, 163, 164 are generated. Resultant object detection data161, 162, 163, 164 may include any suitable data or data structuresindicating resultant objects detected in respective ones of frames 114,113, 112, 111, For example, resultant object detection data 161, 162,163, 164 may indicate object detection localization (e.g., a boundingbox index or bounding box location and size), a class (e.g., an index ofone of multiple available object classes: person, building, car, dog,cattle, etc.), and a confidence scoring (e.g., a value between 0 and 1indicating a confidence in the class) for one or more detected objects.

Resultant object detection data 161, 162, 163, 164 may be generatedusing concatenated feature maps 141, 142, 143, 144 using any suitabletechnique or techniques. In an embodiment, concatenated feature maps141, 142, 143, 144 are provided to and processed by one or more neuralnetwork layers to generate resultant object detection data 161, 162,163, 164. For example, the neural network layer(s) may be pretrained asdiscussed further herein to generate resultant object detection data161, 162, 163, 164 based on concatenated feature maps 141, 142, 143,144.

In another embodiment, shared object patches are determined between thereverse feature maps and forward feature maps as provided byconcatenated feature maps 141, 142, 143, 144. For each shared objectpatch (as identified by shared object patch or bounding box indices),the prediction result having the higher confidence is used to generateobject detection data 161, 162, 163, 164 while the lower predictionresult is discarded. For example, if a shared object patch has aprediction result of “dog”, 0.8 in the forward feature map and aprediction result of “car”, 0.4 in the reverse feature map, theprediction result of “dog”, 0.8 is used while the prediction result of“car”, 0.4 is discarded. The retained prediction results for all of theavailable object patches are then used to generate object detection data161, 162, 163, 164. For example, each object patch having a confidencescore above a threshold may be reported in object detection data 161,162, 163, 164, only a top particular number (e.g., one, three, five,etc.) of top scoring object patches may be reported, etc.

In some embodiments, a shared object patch between first and secondenhanced feature maps are determined. As used herein, a shared objectpatch indicates two patches that have the same or similar localizationinformation. Then, object detection localization and confidence scoringare retained from the feature map for the shared object patch (and thelower scoring prediction results are discarded) in response to theobject detection confidence scoring for the shared object patchcomparing favorably to the other object detection confidence scoring. Insome embodiments, object detection localization and confidence scoringfor shared object patches between first and second feature maps areretained based on higher confidence scoring for each of the sharedobject patches to generate a bidirectional enhanced feature map for thecurrent frame. The bidirectional enhanced feature map may then be usedto determine final object detection data. In an embodiment, a softmaxfunction is applied to the bidirectional enhanced feature map todetermine final object detection data. In an embodiment, a pretrainedneural network layer or layers is applied to the bidirectional enhancedfeature map to determine final object detection data.

Object detection data 161, 162, 163, 164 may include any suitable datasuch as object detection localization, class, and confidence scoring. Asdiscussed, reverse or forward processing is completed for a videosequence prior to the opposite direction processing being completed.With reference to FIG. 1 , in some embodiments, prior to modifying aninitial feature map for any frame N−x, enhanced feature maps includingforward object detection localization and confidence scoring (e.g., in atemporal order of the video or opposite the temporal order), aregenerate for all frames 1 through N−1−x. Thereby, for frame N−x, frameN−1−x has been enhanced prior to frame N-x processing. Furthermore,enhanced feature maps including forward object detection localizationand confidence scoring may be generated for all subsequent frames N−x+1though N prior to opposite direction processing. Thereby, all firstdirection processing is finished prior to any second directionprocessing beginning. Each enhanced feature map from the first directionprocessing is the cached prior to any second direction processingbeginning. The second direction processing is then performed in asimilar sequential manner. As each second direction enhanced feature mapis generated, it may be paired with its first direction feature mappartner and used to generate final output object detection data.

FIG. 3 illustrates an example bidirectional pairing architectureinference network 300 for performing object detection or recognition invideo, arranged in accordance with at least some implementations of thepresent disclosure. For example, network 300 may be implemented as anycombination of two still object detectors, a shared similarity detector,a concatenation module, and an election module, as shown in FIG. 1 . Themodules and components discussed with respect to FIG. 3 may supplementthose modules and components discussed with respect FIG. 1 . Network 300may be implemented in device 100 or any other device or system discussedherein.

As shown, network 300 receives video data 301 including a sequence offrames such as frames 111, 112, 113, 114 or any other frames discussedherein. For example, video data 301 may be a video stream of framesreceived from an imaging device (not shown). As shown, the frames ofvideo data 301 may be reversed at reverse layer 302 such that processingis performed first in a first direction (e.g., forward) and subsequentlyin a second direction opposite the first direction. As shown, a pair offrames are established as pair of frames 303, 304 from video data 301such that pair of frames 303, 304 are consecutive frames of video data301 in either a temporal order or an order opposite the temporal order(e.g., pair of frames 303, 304 may be a current frame, i−1, and asubsequent frame, i, or a previous frame, i−1, and a current frame, i).

Network 300 includes a pair of still image object detectors including astill image object detector 305 and a paired still image object detector306. As shown, still image object detector 305 may receive frame i-I andpaired still image object detector 306 may receive frame i (in a forwardprocessing mode) or still image object detector 305 may receive frame iand paired still image object detector 306 may receive frame i−1 (in areverse processing mode). Still image object detectors 305, 306 may beany suitable still image object detectors such as region proposalnetwork (faster R-CNN) detectors, region-based fully convolutionalnetwork (R-FCN) detectors, single-shot multibox (SSD) detectors, or youonly look once (YOLO) detectors. Still image object detector 305, 306may be any pair of still object detectors illustrated in FIG. 1 afterthe training described in further detail herein.

As shown, still image object detectors 305, 306 generate feature maps307, 308, respectively. As discussed, feature map 307 may include one ortwo (or more) feature maps indicating localization information (loc) andconfidence information or scores (conf) for the frame provided to stillimage object detector 305. Similarly, feature map 308 may include one ortwo (or more) feature maps indicating localization information (loc-p)and confidence information or scores (conf-p) for paired frame 304provided to paired still image object detector 306. For example, thelocalization information may include predicted bounding box coordinates(x_(min), y_(min)) and (x_(max), y_(max)) or a predicted bounding boxindex (which may be used to access the coordinates and/or bounding boxsize). The confidence scores may indicate a level of confidence that abounding box is associated with one or more particular objectcategories. For example, the bounding box as indicated by thelocalization information may be a default prior bounding box forone-stage object detector (e.g., SSD, YOLO) or a posterior calculatedbounding box for two-stage object detector (e.g., RFCN). In someembodiments, the confidence scores may be any number of confidencescores associated with a number of possible object types orclassifications. For example, given the possible classifications of“cat,” “dog,” “car,” and “background,” the confidence information maybe: (cat_score, dog_score, car_score, background_score), wherecat_score, dog_score, car_score, and background_score are confidencescores for each of the possible classifications.

Network 300 also includes a similarity detector 309 communicativelycoupled to still image object detectors 305, 306. Similarity detector309 may be any suitable similarity detector to detect similar patchesbetween pair of frames 303, 304. In an embodiment, similarity detector309 is a Siamese-CNN (convolutional neural network). Similarity detector309 may be any similarity detector illustrated in FIG. 1 after thetraining described in further detail herein. Similarity detector 309generates similarity detection 310, which may include any data or datastructures indicating similar patches between frame 303 and paired frame304. For example, similarity detection 310 may provide pairings of patchindices between patches of frames 303, 304 that match (e.g., patch index0301 of frame 303 matches of patch index 2843 of frame).

Network 300 further includes a history max score cache 311communicatively coupled to an enhancement module 220, which is alsocommunicatively coupled to similarity detector 309 to receive thesimilarity detection 310. History max score cache 311 includes a historyof maximum confidence scores for bounding boxes being evaluated.Enhancement module 312 generates an enhanced feature map 312 for frame304 using similarity detection 310, feature map 307, feature map 308,and history max score cache 311 data. As discussed with respect tofeature maps, 307, 308, feature map 313 may include one or two (or more)feature maps indicating localization information (loc-p) and confidenceinformation or scores (conf-p) for frame 304 after feature map 308 isenhanced as discussed herein.

For example, similarity detector 309 detects paired object patches inframes 303, 304 based on a detected similarity between the pairedobjects. For example, similarity detector 309 may be trained using acontrastive loss based on Euclidean distance as described herein belowto compare bounding boxes between frames 303, 304 to determine theirsimilarity. In some embodiments, if two paired bounding boxesdemonstrate similarity above a predetermined threshold (e.g., 0.5), thenan indicator in similarity detection 310 is generated for the pairedbounding boxes or patches. Enhancement module 312 can modify theprediction result within feature map 308 for frame 304 using theprediction result for the similar patch in frame 303 (as stored inhistory max score cache 311) as discussed herein. For example, if thesimilar patch in frame 303 has a higher confidence score, the predictionresult for the patch in frame 304 is modified to the higher confidencescore and classification.

For example, enhancement module 312 can generate an enhanced confidencescore for feature map 308 for each detected paired patch as indicated bysimilarity detection 310 in frame 304. Based on the localizationinformation and the confidence scores in feature map 307, the confidencescore for the similar patch in frame 303 is retrieved and compared withthe confidence score of the patch in frame 304. When the confidencescore in feature map 307 is greater than the confidence score in featuremap 308, the confidence score and classification for the localizationinformation of the patch is updated to generate enhanced feature map313.

For example, if history max score cache 311 includes a prediction resultwith a greater confidence value for a similar patch (a max in thehistory) of any given patch, then the prediction result may be changedto the prediction result in history max score cache 311 (by changing theconfidence score and classification but not the localizationinformation). For example, history max score cache 311 can store theenhanced confidence score and localization information for each patch asframes are processed. Therefore, feature map 313 includes maximumavailable confidence scores for each patch from feature map 308 andhistory max score cache 311.

As shown, network 300 includes a judgment module 314. If there is notany output from a reverse layer, judgment module 314 only selectsfeature map 313 for later utilization and feature map 313 is cached forlater use in feature map cache 315. The discussed processing is thenrepeated for any number of frames in the forward direction.Subsequently, the discussed processing is repeated in the reversedirection to generate a feature map in analogy to feature map 313 (e.g.,feature map 313 is a forward enhanced feature map and the subsequentfeature map is a reverse enhanced feature map). Then, judgment module314 selects the current enhanced feature map (e.g., reverse) and thepreviously cached enhanced feature map (e.g., forward) via feature mapcache 315 and provides them to concatenation layer 316 for processing.

Concatenation layer 316 may be implemented via any of concatenationmodules 131, 132, 133, 134 illustrated in FIG. 1 after the trainingdescribed in further detail herein. Concatenation layer 316 concatenatesthe enhanced feature maps and provides them to election layer 317.Election layer 317 may be implemented via any of election modules 151,152, 153, 154 illustrated in FIG. 1 after the training described infurther detail herein. Election layer 317 generates resultant objectdetection data 318 using the concatenated feature maps. Resultant objectdetection data 318 may include any suitable data or data structuresindicating resultant object detection. In some embodiments, electionlayer 317 applies one or more pretrained neural network layers to theconcatenated feature maps to generate object detection data 318. In someembodiments, election layer 317 provides maximum confidence valueprediction results for shared object patches between the concatenatedfeature maps.

As discussed, feature maps may be generated for an input frame using astill object detector and the resultant feature maps may be enhancedusing previous frame information (in a first direction of processing)and using subsequent frame information (in a second direction ofprocessing opposite the first direction). The enhanced feature maps maythen be concatenated and processed to determine resultant objectdetection data.

FIG. 4 illustrates example bounding boxes or patches of an example frame400, arranged in accordance with at least some implementations of thepresent disclosure. For frame 400, any number of candidate or defaultbounding boxes may be established, a small number of which areillustrated with respect to bounding boxes 401-407. Notably, for each ofthe candidate bounding boxes (e.g., thousands of candidate boundingboxes), localization information and confidence information aregenerated using still image object detection and enhanced as discussedherein. Such processing is generally illustrated by operation 410. Asshown, feature map 411 may include a confidence feature map 412 and alocalization feature map 413. As discussed herein, localization featuremap 413 may indicate localization information for the candidate boundingboxes and confidence feature map 412 indicates object detectionclassification and scoring for the candidate bounding boxes.

Furthermore, as discussed with respect to enhancement module 312, forbounding boxes of a current frame, prediction results may be enhancedusing patch similarity techniques and/or history max score techniques.

FIG. 5 is a diagram of exemplary local searching pairing for adjacentframes to a current frame, arranged in accordance with at least someimplementations of the present disclosure. As shown, for a current frame501, the frame may be paired to a previous frame 502 during forwardprocessing and to a subsequent frame 503 during reverse processing.Current frame 501 includes a default bounding box 511 and additionalbounding boxes 512 (all lose not labeled A0). Previous frame 502includes a default bounding box 521 in the same position as defaultbounding box 511 of current frame 501 (e.g., they are collocated).Previous frame 502 also includes a second set of default bounding boxes522 that border bounding box 521. Previous frame 502 further includes athird set of default bounding boxes 523 that border second set ofdefault bounding boxes 522. Arrows 505, 506, 507 indicate a pairing ofdefault bounding box 511 with default bounding box 521 second set ofdefault bounding boxes 522, and third set of default bounding boxes 523,respectively.

As shown, in forward processing, default bounding boxes that infer thehighest confidence scores for confidence information in current framemay be determined. For default bounding box 511, it may be paired withdefault bounding boxes including default bounding boxes 521, 522, 523.For example, default bounding box 511 may be paired with 9 or moredefault bounding boxes in previous frame 502 including the defaultbounding box 521 and the second set of default bounding boxes 522. Insome embodiments, default bounding box 511 can be paired with 25bounding boxes in previous frame 502 including the default bounding box521, second set of default bounding boxes 522, and third set of defaultbounding boxes 523. The resulting pairs may include (A0, a0), (A0, b0),(A0, b1), . . . (A0, b7), (A0, c0), (A0, c1), (A0, c15). The sets ofpairs can be used by a similarity module to compare and track the sameobject patch, as described in greater detail herein.

Similarly, in reverse processing, default bounding boxes that infer thehighest confidence scores for confidence information in current framemay again be determined. For default bounding box 511, it may be pairedwith default bounding boxes including default bounding boxes 531, 532,533, For example, default bounding box 511 may be paired with 9, 25, ormore default bounding boxes in subsequent frame 503 (as shown via arrows508, 509, 510) including a default bounding box 531, a second set ofdefault bounding boxes 532 and/or a third set of default bounding boxes533. The sets of pairs again can be used by a similarity module tocompare and track the same object patch

FIG. 6 illustrates an example bidirectional pairing architecturetraining network 600 for performing object detection or recognition invideo, arranged in accordance with at least some implementations of thepresent disclosure. For example, training network 300 may be used topretrain network 300 for implementation in device 100. As used herein,the term true positives (TP) indicates positive samples correctlylabeled by the classifier, the term true negatives (TN) indicatesnegative samples that were correctly labeled by the classifier, falsepositives (FP) indicate negative samples that were incorrectly labeledas positive, and false negatives (FN) indicate positive samples thatwere mislabeled as negative.

As shown, training network 300 receives video data 601 includingmultiple sequence of frames in analogy to frames 111, 112, 113, 114. Forexample, video data 601 may include multiple video streams of framesreceived as a training corpus to provide pretraining of system forobject detection or recognition in video. For example, video data 601may be training data including one or more objects within frames suchthat the objects are at least partially occluded.

As shown, the frames of video data 601 may be reversed at reverse layer602 such that processing is performed first in a first direction (e.g.,forward) and subsequently in a second direction opposite the firstdirection. As shown, a pair of frames are established as pair of frames603, 604 from video data 601 such that pair of frames 603, 604 areconsecutive frames of a sequence of video data 601 in either a temporalorder or an order opposite the temporal order (e.g., pair of frames 603,604 may be a current frame, i−1, and a subsequent frame, i, or aprevious frame, i−1, and a current frame, i).

Training network 300 includes a pair of still image object detectorsincluding a still image object detector 605 and a paired still imageobject detector 606. As shown, still image object detector 605 receivesframe i−1 and paired still image object detector 606 receives frame i(in a forward processing mode) or still image object detector 305receives frame i and paired still image object detector 306 receivesframe i−1 (in a reverse processing mode). Still image object detectors305, 306 may be any suitable still image object detectors such as regionproposal network (faster R-CNN) detectors, region-based fullyconvolutional network (R-FCN) detectors, single-shot multibox (SSD)detectors, or you only look once (YOLO) detectors. Still image objectdetector 305, 306 may be any pair of still object detectors illustratedin FIG. 1 after the training described in further detail herein.

As shown, still image object detectors 605, 606 generate feature maps607, 608, respectively. As discussed, feature map 607 may include one ortwo (or more) feature maps indicating localization information (loc) andconfidence information or scores (conf) for the frame provided to stillimage object detector 605. Similarly, feature map 608 may include one ortwo (or more) feature maps indicating localization information (loc-p)and confidence information or scores (conf-p) for paired frame 604provided to paired still image object detector 606.

Training network 300 further includes a similarity detector 609, whichmay be any suitable similarity detector to detect similar patchesbetween pair of frames 603, 604. In an embodiment, similarity detector609 is a Siamese-CNN. Similarity detector 609 determines similar patchesbetween frame 603 and paired frame 604. Training network 300 furtherincludes an enhancement module 312, which generates an enhanced featuremap 612 for frame 604 using detected similarity, feature map 607, andfeature map 608. For example, detected paired object patches (based on adetected similarity) between frames 603, 604 may be trained usingsimilarity detector 609 based on a contrastive loss based on Euclideandistance. Enhancement module 612 can then modify the prediction resultwithin feature map 608 for frame 604 using the prediction result for thesimilar patch in frame 603 as discussed herein. Enhancement module 612may also modify a prediction result within feature map 608 using ahistory max score cache as discussed herein.

If there is not any output from a reverse layer, a judgment module 614selects feature map 613 for later utilization and feature map 613 iscached for later use. The discussed processing is then repeated for anynumber of frames in the forward direction. Subsequently, the discussedprocessing is repeated in the reverse direction to generate feature mapsin analogy to feature map 613 for the reverse direction. Judgment module614 then selects the current enhanced feature map (e.g., reverse) andthe previously cached enhanced feature map (e.g., forward) and providesthem to concatenation layer 616. Concatenation layer 616 concatenatesthe enhanced feature maps and provides them to election layer 617, whichgenerates resultant object detection data by applying one or morepretrained neural network layers to the concatenated feature maps orproviding maximum confidence value prediction results for shared objectpatches between the concatenated feature maps.

Training network 300 includes a localization (loc) and confidence (conf)loss calculator 618, a contrastive loss calculator 619, and alocalization (loc) and confidence (conf) loss calculator 620. Trainingnetwork 300 further includes a multi-loss calculator 621 coupled tolocalization and confidence loss calculator 618, contrastive losscalculator 619, and localization and confidence loss calculator 620. Thetraining objective of training network 300 may be to minimize the sum ofa similarity loss between two object patches (contrastive loss), thelocalization loss for object bounding box detection (loc loss), and theconfidence loss for object classification (conf loss). In someembodiments, before training is performed, default bounding boxescorresponding to a ground truth detection may first be determined orestablished. As used herein, a default bounding box indicates a kind ofpatch proposal policy used to determine a detection region for an object(e.g., proposed regions for detection). Default bounding boxes may alsobe referred to as default boxes, prior boxes, or anchor boxes. In someembodiments, a default bounding box having a higher Jaccard overlap withthe ground truth bounding boxes may be referred to as a positive defaultbounding box. A ground truth bounding box is a bounding box labeled withknown true information in the dataset. For example, positive defaultbounding boxes may have a Jaccard overlap with the ground truth boundingbox that is more than a predetermined threshold (e.g., 0.5). Otherwise,the default bounding box is negative. The Jaccard overlap J may becalculated using Equation (1):

$\begin{matrix}{{J\left( {S_{{box}1},S_{box2}} \right)} = \frac{S_{box1}\bigcap S_{box2}}{S_{box1}\bigcup S_{box2}}} & {{Equation}(1)}\end{matrix}$where S is the area of each respective bounding box, and box 1 is thedefault bounding box and box2 is the ground truth bounding box. Groundtruth bounding boxes, as used herein, refer to a label of a bounding boxin the dataset, and may also be referred to as the ground truth box.Thus, negative default bounding boxes may have a Jaccard overlap withground truth bounding box that is less than the predetermined threshold.

In an embodiment, the training objective can be based on the training ofSSD networks. For example, x^(p) _(ij)={1, 0} and y^(p) _(ij)={1, 0} maybe indicators for matching the i-th default bounding box to the j-thground truth bounding box of category p in the previous and referenceframe. Given predicted boxes in the previous and reference frame ofI_(f-1) and l_(f), and ground truth bounding boxes in the previous andreference frame of g_(f-1) and g_(f); and corresponding default boundingboxes in the previous and reference frame of d_(f-1) and d_(f), then theoverall objective loss function L can be calculated by the multi-losscalculator 621 as a weighted sum of the localization loss (loc loss),the contrastive loss (contra loss) and the confidence loss (conf loss)using Equation (2):

$\begin{matrix}{{L\left( {x,y,\ c,\ e,l_{f - 1},l_{f},g_{f - 1},g_{f},d_{f - 1},d_{f}} \right)} = {\left( {{L_{conf}\left( {x,c} \right)} + {\alpha*{L_{loc}\left( {x,{l_{f} - 1},g_{f - 1}} \right)}}} \right) + {\frac{1}{N}\left( {{L_{conf}\left( {y,e} \right)} + {\alpha*{L_{loc}\left( {y,l_{f},g_{f}} \right)}}} \right)} + {\frac{1}{\theta MN}\beta*{L_{contra}\left( {d_{f - 1},d_{f}} \right)}}}} & {{Equation}(2)}\end{matrix}$where NI and N are the numbers of matched default boxes in the previousand reference frame. If M=0 and N !=0, then the trainer may onlyconsider L_(conf) and L_(loc) for the reference frame. If N=0 and M !=0,then the trainer may only consider L_(conf) and L_(loc) for the previousframe. If M=0 and N=0, then the trainer may set the overall loss to 0.In some examples, the localization loss can be a smooth L1 loss betweenthe predicted box (l) and the ground truth bounding box (g) parameters.

In some examples, since L_(loc)(x, lf−1, gf−1) and L_(loc)(y, lf, gf))are the same formula, while L_(conf)(x, c) and L_(conf)(y, e) are alsothe same formula, L_(loc)(x, l, g) can be calculated using Equation (3):

$\begin{matrix}{{{L_{loc}\left( {x,\ l,\ g} \right)} = {{\sum\limits_{i \in {Pos}}^{N}{\sum\limits_{m \in {\{{{cx},{cy},w,h}\}}}{x_{ij}^{k}{{smooth}_{L1}\left( {l_{i}^{m} - {\hat{g}}_{j}^{m}} \right)}}}} + \underset{i \in {Neg}}{0}}}{{\hat{g}}_{j}^{cx} = {\left( {g_{j}^{cx} - d_{i}^{cx}} \right)/d_{i}^{w}}}{{\hat{g}}_{j}^{cy} = {\left( {g_{j}^{cy} - d_{i}^{cy}} \right)/d_{i}^{h}}}{{\hat{g}}_{j}^{w} = {\log\left( {g_{j}^{w}/d_{i}^{w}} \right)}}{{\hat{g}}_{j}^{h} = {\log\left( {g_{j}^{h}/d_{i}^{h}} \right)}}} & {{Equation}(3)}\end{matrix}$

Similarly, L_(conf)(x, c) can be calculated using Equation (4):

$\begin{matrix}{{{L_{conf}\left( {x,c} \right)} = {{- {\sum\limits_{i \in {Pos}}^{N}{x_{ij}^{p}{\log\left( {\hat{c}}_{i}^{p} \right)}}}} - {\sum\limits_{i \in {Neg}}^{N}{\log\left( {\hat{c}}_{i}^{0} \right)}}}}{where}{\left( {\hat{c}}_{i}^{p} \right) = \frac{\exp\left( c_{i}^{p} \right)}{\sum_{p}{\exp\left( c_{i}^{p} \right)}}}} & {{Equation}(4)}\end{matrix}$

The contrastive loss is the loss of the corresponding default boundingboxes between two adjacent frames, and can be calculated using Equation(5):

$\begin{matrix}{{L_{contra}\left( {d_{f - 1},d_{f}} \right)} = {\sum\limits_{n = 1}^{\theta{MN}}\left( {{YD}^{2} + {\left( {1 - Y} \right){\max\left( {{{threshold} - D},0} \right)}^{2}}} \right)}} & {{Equation}(5)}\end{matrix}$where D=∥^(d) _(f-1)−d_(f)∥₂, which is the Euclidean distance of defaultbounding boxes between two adjacent frames, Y is the label whether twoboxes are paired or not, threshold is the threshold of two boxes whichare unpaired, and θ is the proportion of paired boxes and unpairedboxes. The weight terms α and β can be used to perform cross validation.

As discussed, after pretraining, components of training network 300 areimplemented as components of network 300.

FIG. 7 is a flow chart illustrating a process 700 for training abidirectional pairing architecture to perform object detection orrecognition in video neural network to detect objects in video frames,arranged in accordance with at least some implementations of the presentdisclosure. For example, process 700 may be performed by trainingnetwork 300 or any other system or device discussed herein.

Process 700 begins at operation 701, where a video sequence from atraining data set is received. For example, the video sequence mayinclude various objects in which some are partially obscured. Processingcontinues at operation 702, where a determination is made as to whetherto reverse the frames of the video sequence. For example, operations703-706 may be performed for the sequence in a first order and thenperformed in a second order opposite the first order. At operation 707,when the first order processing is complete, the resultant feature mapsare cached and, when the second order processing is complete, theresultant feature maps are concatenated with the first resultant featuremaps for each frame.

Processing continues at operation 703 where, in either the first orsecond order, still image object detection is performed on a first frameof the video frames to generate a feature map including a localizationfeature map and a confidence feature map and at operation 704 wherestill image object detection is performed on a next (second) frame ofthe video frames (in the selected order) to generate a paired featuremap including a localization feature map and a confidence feature map.

Processing continues at operation 705 where similarity detection isperformed between the first and second frames to generate similaritydetection data indicative of similar patches between the first andsecond frames. Processing continues at operation 706 where the featuremap for the second frame is enhanced using the feature map for the firstfame based on the similar patches. For example, any patches in thesecond frame having confidence values less than their respective patchesin the first frame have their prediction results updated based on theprediction result of the paired patch in the first frame.

As discussed, at operation 707, after processing in the first direction,the enhanced feature map for the frame is cached. Furthermore,operations 703-706 are performed for each pairing of frames sequentiallyin the first order to generate an enhanced feature map for each frame,which may be characterized as forward enhanced feature maps.Subsequently, the order of the frames is reversed and such processing isrepeated in the second direction, opposite the first direction, togenerate a second enhanced feature map for each frame, which may becharacterized as reverse enhanced feature maps. After such oppositedirection processing, operation 707 is used to concatenate, for eachframe, the forward enhanced feature map and the reverse enhanced featuremap to determine a concatenated feature map for each frame.

Processing continues at operation 708, where the concatenated featuremap for each frame is provided to an election layer to generate a finalfeature map and/or final object detection result data. As discussed, theelection layer may be one or more neural network layers or the electionlayer may determine, for each shared patch of the concatenated featuremap, a higher scoring prediction result, retain the higher scoringprediction result, and discard the lower scoring prediction result.

Processing continues at operation 709, where the localization loss andconfidence loss for the first frame still image detection, contrastiveloss for the similarity detector, and the localization loss andconfidence loss for final feature map generation (e.g., from operation708) are determined and summed to generate a multi-loss measure for thediscussed implementation. In some embodiments, the localization loss andconfidence loss determinations include matching default bounding boxesto ground truth bounding boxes to generate positive default boundingboxes and negative default bounding boxes, determining a classconfidence score for each positive default bounding box and a backgroundconfidence score for each negative default bounding box, and determininga localization gap between the positive default bounding boxes and theground truth bounding boxes.

Processing continues at operation 710, where a gradient is determinedbased on the multi-loss measurement and used to backward propagate themulti-loss measurement. For example, the gradient may be calculatedusing partial derivatives with respect to each of the parameters of themulti-loss function. In some examples, the gradient based on themulti-loss measurement is propagated backwards through the entiretraining network or architecture, including the similarity detector,still image object detectors, and the election layer.

Processing continues at decision operation 711, where a determination ismade as to whether a convergence is detected. If so, process 700 ends atend operation 712. If not, processing continues at operation 701 whereanother sequence of video frames for training is received and processingcontinues as discussed until convergence is met. The convergencedecision may be based on the multi-loss measurement meeting a threshold,a maximum number of iterations being reached, etc.

FIG. 8 is a flow chart illustrating a process 800 for performing objectdetection or recognition in video using a bidirectional pairingarchitecture, arranged in accordance with at least some implementationsof the present disclosure. For example, process 700 may be performed bynetwork 300, device 100, or any other system or device discussed herein.

Process 800 begins at operation 801, where a video sequence is received.The video sequence may include any number of frames and may be receivedfrom a video camera, a memory, another device, or any other suitablecomponent or device. Processing continues at operation 802, where adetermination is made as to whether to reverse the frames of the videosequence. For example, operations 803-806 may be performed for thesequence in a first order and then performed in a second order oppositethe first order. At operation 807, when the first order processing iscomplete, the resultant feature maps are cached and, when the secondorder processing is complete, the resultant feature maps areconcatenated with the first resultant feature maps for each frame.

Processing continues at operation 803 where, in either the first orsecond order, still image object detection is performed on a first frameof the video frames to generate a feature map including a localizationfeature map and a confidence feature map and at operation 804 wherestill image object detection is performed on a next (second) frame ofthe video frames (in the selected order) to generate a paired featuremap including a localization feature map and a confidence feature map.As with operations 703, 704, the still image object detectors maygenerate potential object patches and perform object detection for theobject patches. For example, the still image object detectors mayperform a patch proposal to generate patches (or candidate boundingboxes). The method used for patch proposal may be based on the type ofstill image object detector used, including SSD, YOLO, Faster R-CNN, orR-FCN, among other possible still image object detectors. In someembodiments, the prior proposal method may include generating a fixednumber and position of patches such as 8,732 patch proposals for eachframe with the patches covering different areas, different aspectratios, and different scale sizes. In some embodiments, a posteriorproposal method may be used with a two-stage object detector, such as anR-FCN detector. The posterior proposal method may generate new patchesfor each sample first, and then perform confidence prediction based onnew patches and similarity detection based on new paired patches. Forexample, a selective search algorithm can be utilized to perform patchproposal for a two-stage object detector such that the selective searchalgorithm generates object localization hypotheses by picture contentthat are used to generate posterior proposal patches, which are thenused for object detection prediction.

Processing continues at operation 805 where similarity detection isperformed between the first and second frames to generate similaritydetection data indicative of similar patches between the first andsecond frames. Processing continues at operation 806 where the featuremap for the second frame is enhanced using the feature map for the firstfame based on the similar patches. For example, any patches in thesecond frame having confidence values less than their respective patchesin the first frame have their prediction results updated based on theprediction result of the paired patch in the first frame. As withoperation 706, such enhancement may be performed as follows. First, allthe default bounding boxes that successfully infer a higher confidencescore for confidence info in the second frame may be used to generate afirst paired patch or box in the second frame. Then, a second paired boxin the first frame is determined such that the first and second pairedboxes are similar. For example, a patch or default bounding box A₀ inthe second frame may pair with any of 9 (8+1), 25 (16+8+1) or morepatches or default bounding boxes in the first frame, as discussedherein. For example, the similar paired patches may include the sameobject, but the appearance of the object may differ somewhat due tomotion blur, partial occlusion, being in a different pose, etc. Next, aconfidence score for the second frame may be modified based onlocalization information and/or a history max score cache.

In some embodiments, a history max score cache is accessed using thelocalization information to verify that the confidence score for thesecond frame is inherited from a historical best detection result for acollocated patch. For example, an index for a patch in the second framemay be 8721 and the patch may have a confidence score from the stillimage object detector of 0.45 for a cat category (8721, 0.45, cat). Ifthe history max score cache includes a confidence score for the patchindex 8721 of 0.78 for the dog category (8721, 0.78, dog), then theconfidence score can be modified from (8721, 0.45, cat) to (8721, 0.78,dog) in the feature map for the second frame.

In addition or in the alternative, the feature map for the second frameis enhanced using one or more paired confidence scores from the firstframe based on the similarity detection. In some embodiments, theprediction result from the paired still image object detector for thesecond frame is enhanced by using the similarity detection and comparingthe paired confidence score with the confidence score of thecorresponding confidence score of the paired patch in the first frame.For example, an enhanced confidence score in the enhanced feature mapmay be used for each detected paired object patch in the second framebased on a higher confidence score from the feature map (for the pairedpatch) for the first frame. In an embodiment, the confidence scorescorresponding to the patches are compared and the prediction result forthe second frame is only updated when the confidence score for the patchin the first frame exceeds the confidence score for the patch in thesecond frame. For example, a similarity detection result may indicate apatch of index 8721 in the second frame is paired with a patch of index6618 in the first frame. Furthermore, the prediction result for patch8721 may be (8721, 0.39, cat) and the prediction result for patch 6618may be (8721, 0.78, dog). Notably, the prediction result for patch 6618of the first frame may have been previously enhanced or it may be from astill image object detector. The confidence score of 0.78 (8721, 0.78,dog) is compared with the paired confidence score of 0.39 (6618, 0.39,cat). Since 0.78 is higher than 0.39, the former (dog) wins, and thefeature map is enhanced by changing the prediction result therein from(6618, 0.39, cat) to (6618, 0.78, dog). In this example, 6618 is thedefault bounding box localization index in the second frame. Notably,the localization index does not change while the confidence score andclassification (0.78, dog are changed.

As discussed, at operation 807, after processing in the first direction,the enhanced feature map for the frame is cached. Furthermore,operations 803-806 are performed for each pairing of frames sequentiallyin the first order to generate an enhanced feature map for each frame,which may be characterized as forward enhanced feature maps.Subsequently, the order of the frames is reversed and such processing isrepeated in the second direction, opposite the first direction, togenerate a second enhanced feature map for each frame, which may becharacterized as reverse enhanced feature maps. After such oppositedirection processing, operation 807 is used to concatenate, for eachframe, the forward enhanced feature map and the reverse enhanced featuremap to determine a concatenated feature map for each frame.

Processing continues at operation 808, where the concatenated featuremap for each frame is provided to an election layer to generate a finalfeature map and/or final object detection result data. As discussed, theelection layer may be one or more neural network layers or the electionlayer may determine, for each shared patch of the concatenated featuremap, a higher scoring prediction result, retain the higher scoringprediction result, and discard the lower scoring prediction result.

FIG. 9 is a flow diagram illustrating an example process 900 forperforming object detection, arranged in accordance with at least someimplementations of the present disclosure. Process 900 may include oneor more operations 901-905 as illustrated in FIG. 9 . Process 900 mayform at least part of an object recognition or detection process. By wayof non-limiting example, process 900 may form at least part of an objectrecognition or detection process performed by device 100 as discussedherein during an implementation phase. Furthermore, process 900 will bedescribed herein with reference to system 1000 of FIG. 10 ,

FIG. 10 is an illustrative diagram of an example system 1000 forperforming object detection, arranged in accordance with at least someimplementations of the present disclosure. As shown in FIG. 10 , system1000 may include one or more central processing units (CPU) 1001, agraphics processing unit 1002, and memory stores 1003. Also as shown,graphics processing unit 1002 may include one or more still image objectdetectors 101, 102, one or more similarity detectors 105, one or moreconcatenate modules 131, and one or more election modules 151. Suchmodules may be implemented to perform operations as discussed herein. Inthe example of system 1000, memory stores 1003 may store input framedata, feature maps, enhanced feature maps, concatenated feature maps,resultant object detection data, localization information, confidenceinformation, or any other data or data structure discussed herein.

As shown, in some examples, one or more still image object detectors101, 102, one or more similarity detectors 105, one or more concatenatemodules 131, and one or more election modules 151 are implemented viagraphics processing unit 1002. In other examples, one or more orportions of one or more still image object detectors 101, 102, one ormore similarity detectors 105, one or more concatenate modules 131, andone or more election modules 151 are implemented via central processingunits 1001 or an image processing unit (not shown) of system 1000. Inyet other examples, one or more or portions of one or more still imageobject detectors 101, 102, one or more similarity detectors 105, one ormore concatenate modules 131, and one or more election modules 151 areimplemented via an imaging processing pipeline, graphics pipeline, orthe like.

Graphics processing unit 1002 may include any number and type ofgraphics processing units that may provide the operations as discussedherein. Such operations may be implemented via software or hardware or acombination thereof. For example, graphics processing unit 1002 mayinclude circuitry dedicated to manipulate frame data, object detectionarchitecture data, etc. obtained from memory stores 1003. Centralprocessing units 1001 may include any number and type of processingunits or modules that may provide control and other high level functionsfor system 1000 and/or provide any operations as discussed herein.Memory stores 1003 may be any type of memory such as volatile memory(e.g., Static Random Access Memory (SRAM), Dynamic Random Access Memory(DRAM), etc.) or non-volatile memory (e.g., flash memory, etc.), and soforth. In a non-limiting example, memory stores 1003 may be implementedby cache memory. In an embodiment, one or more or portions of one ormore still image object detectors 101, 102, one or more similaritydetectors 105, one or more concatenate modules 131, and one or moreelection modules 151 are implemented via an execution unit (EU) ofgraphics processing unit 1002. The EU may include, for example,programmable logic or circuitry such as a logic core or cores that mayprovide a wide array of programmable logic functions. In an embodiment,one or more or portions of one or more still image object detectors 101,102, one or more similarity detectors 105, one or more concatenatemodules 131, and one or more election modules 151 are implemented viadedicated hardware such as fixed function circuitry or the like. Fixedfunction circuitry may include dedicated logic or circuitry and mayprovide a set of fixed function entry points that may map to thededicated logic for a fixed purpose or function. In some embodiments,one or more or portions of one or more still image object detectors 101,102, one or more similarity detectors 105, one or more concatenatemodules 131, and one or more election modules 151 are implemented via anapplication specific integrated circuit (ASIC). The ASIC may include anintegrated circuitry customized to perform the operations discussedherein.

Returning to discussion of FIG. 9 , process 900 begins at operation 901,where still image object detection is performed on each of a current, aprevious, and a subsequent frame of video to determine initial featuremaps including object detection localization and confidence scoring foreach of multiple potential object patches for the frames. For example,an initial feature map is generated for each of the current frame, theprevious frame, and the subsequent frame, which may be characterized asa current frame initial feature map, a previous frame initial featuremap, and a previous frame initial feature map, respectively. The stillimage object detection may be performed using any suitable technique ortechniques such as region proposal network (faster R-CNN) detection,region-based fully convolutional network (R-FCN) detection, single-shotmultibox (SSD) detection, or you only look once (POLO) detection. Thelabels current, previous, and subsequent for the frames is meant toindicate an order of processing that may match a temporal order of theframes or may be opposite the temporal order of the frames. Notably, thediscussed techniques may first be performed in temporal order and then areversed order or vice versa.

Processing continues at operation 902, where, based on patch similarity,first and second paired patches between the current and previous frames,respectively, and third and fourth paired patches between the currentand subsequent frames, respectively, are detected. Such patch matchingbased on patch similarity may be performed using any suitable techniqueor techniques such as applying a pretrained neural network or the like.In an embodiment, detecting the first and second paired patches and/orthe third and fourth paired patches includes implementing a Siameseconvolutional neural network.

Processing continues at operation 903, where a first prediction resultin the current frame initial feature map for the first paired patch ofthe current frame is modified to a maximum cached confidence score ofthe second paired patch of the previous frame to generate a firstenhanced feature map including forward object detection localization andconfidence scoring for the current frame. The first prediction resultmay include any suitable object detection prediction information such asa confidence score and a class label. Notably, the localizationinformation (e.g., bounding box or patch location are not updated). Thefirst enhanced feature map may be cached for later use in objectdetection as is discussed further below. In some embodiments, modifyingthe first prediction result to the maximum cached confidence score is inresponse to a confidence scoring of the maximum cached confidence scoreexceeding a confidence scoring of the first prediction result.

Notably, the maximum cached confidence score is from a previouslyenhanced feature map for the previous frame (and not from the initialfeature map generated at operation 901). In some embodiments, process900 further includes sequentially generating, prior to said modifyingthe first prediction result and in a temporal order of the video, aplurality of enhanced feature maps comprising forward object detectionlocalization and confidence scoring for each of a plurality of frames ofthe video previous to the previous frame and previous frame such thatthe maximum cached confidence score is in the enhanced feature map forthe previous frame. In other embodiments, the forward object detectionmay be in an order of the frames opposite the temporal order of thevideo. In some embodiments, process 900 further includes sequentiallygenerating, subsequent to said modifying the first prediction result andin the temporal order of the video, a second plurality of enhancedfeature maps including forward object detection localization andconfidence scoring for the subsequent frame and each of a plurality offrames of the video subsequent to the subsequent frame. Notably, thediscussed enhancements to the initial feature maps may be madesequentially to video frames in a particular (first) order to generate asequence of enhanced feature maps, each taking advantage of previouslymade enhancements.

Processing continues at operation 904, where a third prediction resultin the current frame initial feature map for the third paired patch ofthe current frame is modified to a maximum cached confidence score ofthe fourth object patch of the subsequent frame to generate at least asecond enhanced feature map including reverse object detectionlocalization and confidence scoring for the current frame. Notably,operation 904 may enhance the initial feature map for the current frameusing enhancements generated by processing frames in a directionopposite to that applied at operation 903. As with the first predictionresult, the third prediction result may include any suitable objectdetection prediction information such as a confidence score and a classlabel (and the localization information is not updated). The secondenhanced feature map may be used immediately and may not need to becached.

Processing continues at operation 905, where object detectioninformation (e.g., an object detection localization, class, andconfidence scoring) for the current frame are determined and outputusing the first and second feature maps. The object detectioninformation may include any suitable object detection information suchas an object detection localization (e.g., bounding box location), class(e.g., semantic class of the object expected to be in the bounding box),and confidence scoring (e.g., a score indicating a confidence the objectis in the bounding box) for any number of detected objects or for one ormore (even all) default bounding boxes.

The object detection information may be determined using any suitabletechnique or techniques. In an embodiment, determining the objectdetection information includes concatenating the first and secondenhanced feature maps (e.g., the forward and reverse enhanced featuremaps) and providing the concatenated first and second enhanced featuremaps to at least one neural network layer to generate the objectdetection information. In an embodiment, determining the objectdetection information includes determining a shared object patch betweenthe first and second feature maps (e.g., a patch having the same index,the same default patch, etc.), retaining only object detectionlocalization and confidence scoring from the first feature map for theshared object patch and discarding object detection localization andconfidence scoring from the second feature map for the shared objectpatch in response to the object detection confidence scoring for theshared object patch in the first feature map comparing favorably to theobject detection confidence scoring for the shared object patch in thesecond feature map (e.g., keeping the prediction result for the higherconfidence score between the two), and determining the object detectioninformation based at least on the retained object detection localizationand confidence scoring from the first feature map. In an embodiment, afeature map of combined highest scores may be provided to a neuralnetwork layer, a softmax function, or the like to generate the objectdetection information. In an embodiment, determining the objectdetection information includes retaining only object detectionlocalization and confidence scoring for shared object patches betweenthe first and second feature maps based on higher confidence scoring foreach of the shared object patches to generate a bidirectional enhancedfeature map for the current frame and determining the object detectioninformation based on the bidirectional enhanced feature map. In anembodiment, determining the object detection information based on thebidirectional enhanced feature map includes applying a softmax functionto the bidirectional enhanced feature map. In an embodiment, determining the object detection information based on the bidirectionalenhanced feature map includes applying at least one neural network layerto the bidirectional enhanced feature map.

Process 900 may provide for generating object detection data or objectlabel data based on a video sequence of input frames. Process 900 may berepeated any number of times either in series or in parallel for anynumber of video sequences. As discussed, process 900 may provide forhigh quality object recognition results.

Various components of the systems described herein may be implemented insoftware, firmware, and/or hardware and/or any combination thereof. Forexample, various components of devices or systems discussed herein maybe provided, at least in part, by hardware of a computingSystem-on-a-Chip (SoC) such as may be found in a computing system suchas, for example, a computer, a laptop computer, a tablet, or a smartphone. For example, such components or modules may be implemented via amulti-core SoC processor. Those skilled in the art may recognize thatsystems described herein may include additional components that have notbeen depicted in the corresponding figures.

While implementation of the example processes discussed herein mayinclude the undertaking of all operations shown in the orderillustrated, the present disclosure is not limited in this regard and,in various examples, implementation of the example processes herein mayinclude only a subset of the operations shown, operations performed in adifferent order than illustrated, or additional operations.

In addition, any one or more of the operations discussed herein may beundertaken in response to instructions provided by one or more computerprogram products. Such program products may include signal bearing mediaproviding instructions that, when executed by, for example, a processor,may provide the functionality described herein. The computer programproducts may be provided in any form of one or more machine-readablemedia. Thus, for example, a processor including one or more graphicsprocessing unit(s) or processor core(s) may undertake one or more of theblocks of the example processes herein in response to program codeand/or instructions or instruction sets conveyed to the processor by oneor more machine-readable media. In general, a machine-readable mediummay convey software in the form of program code and/or instructions orinstruction sets that may cause any of the devices and/or systemsdescribed herein to implement at least portions of the discussedoperations, modules, or components discussed herein.

As used in any implementation described herein, the term “module” refersto any combination of software logic, firmware logic, hardware logic,and/or circuitry configured to provide the functionality describedherein. The software may be embodied as a software package, code and/orinstruction set or instructions, and “hardware”, as used in anyimplementation described herein, may include, for example, singly or inany combination, hardwired circuitry, programmable circuitry, statemachine circuitry, fixed function circuitry, execution unit circuitry,and/or firmware that stores instructions executed by programmablecircuitry. The modules may, collectively or individually, be embodied ascircuitry that forms part of a larger system, for example, an integratedcircuit (IC), system on-chip (SoC), and so forth.

FIG. 11 is an illustrative diagram of an example system 1100, arrangedin accordance with at least some implementations of the presentdisclosure. In various implementations, system 1100 may be a computingsystem although system 1100 is not limited to this context. For example,system 1100 may be incorporated into a personal computer (PC), laptopcomputer, ultra-laptop computer, tablet, phablet, touch pad, portablecomputer, handheld computer, palmtop computer, personal digitalassistant (PDA), cellular telephone, combination cellular telephone/PDA,television, smart device (e.g., smart phone, smart tablet or smarttelevision), mobile internet device (MID), messaging device, datacommunication device, peripheral device, gaming console, wearabledevice, display device, all-in-one device, two-in-one device, and soforth.

In various implementations, system 1100 includes a platform 1102 coupledto a display 1120. Platform 1102 may receive content from a contentdevice such as content services device(s) 1130 or content deliverydevice(s) 1140 or other similar content sources such as a camera orcamera module or the like. A navigation controller 1150 including one ormore navigation features may be used to interact with, for example,platform 1102 and/or display 1120. Each of these components is describedin greater detail below.

In various implementations, platform 1102 may include any combination ofa chipset 1105, processor 1110, memory 1112, antenna 1113, storage 1114,graphics subsystem 1115, applications 1116 and/or radio 1118, Chipset1105 may provide intercommunication among processor 1110, memory 1112,storage 1114, graphics subsystem 1115, applications 1116 and/or radio1118. For example, chipset 1105 may include a storage adapter (notdepicted) capable of providing intercommunication with storage 1114.

Processor 1110 may be implemented as a Complex Instruction Set Computer(CISC) or Reduced instruction Set Computer (RISC) processors, x86instruction set compatible processors, multi-core, or any othermicroprocessor or central processing unit (CPU). In variousimplementations, processor 1110 may be dual-core processor(s), dual-coremobile processor(s), and so forth.

Memory 1112 may be implemented as a volatile memory device such as, butnot limited to, a Random Access Memory (RAM), Dynamic Random AccessMemory (DRAM), or Static RAM (SRAM).

Storage 1114 may be implemented as a non-volatile storage device suchas, but not limited to, a magnetic disk drive, optical disk drive, tapedrive, an internal storage device, an attached storage device, flashmemory, battery backed-up SDRAM (synchronous DRAM), and/or a networkaccessible storage device. In various implementations, storage 1114 mayinclude technology to increase the storage performance enhancedprotection for valuable digital media when multiple hard drives areincluded, for example.

Graphics subsystem 1115 may perform processing of images such as stillimages, graphics, or video for display. Graphics subsystem 1115 may be agraphics processing unit (GPU), a visual processing unit (VPU), or animage processing unit, for example. In some examples, graphics subsystem1115 may perform scanned image rendering as discussed herein. An analogor digital interface may be used to communicatively couple graphicssubsystem 1115 and display 1120. For example, the interface may be anyof a High-Definition Multimedia interface, DisplayPort, wireless LOW,and/or wireless HD compliant techniques. Graphics subsystem 1115 may beintegrated into processor 1110 or chipset 1105. In some implementations,graphics subsystem 1115 may be a stand-alone device communicativelycoupled to chipset 1105.

The image processing techniques described herein may be implemented invarious hardware architectures. For example, image processingfunctionality may be integrated within a chipset. Alternatively, adiscrete graphics and/or image processor and/or application specificintegrated circuit may be used. As still another implementation, theimage processing may be provided by a general purpose processor,including a multi-core processor. In further embodiments, the functionsmay be implemented in a consumer electronics device.

Radio 1118 may include one or more radios capable of transmitting andreceiving signals using various suitable wireless communicationstechniques. Such techniques may involve communications across one ormore wireless networks. Example wireless networks include (but are notlimited to) wireless local area networks (WLANs), wireless personal areanetworks (WPANs), wireless metropolitan area network (WMANs), cellularnetworks, and satellite networks. In communicating across such networks,radio 1118 may operate in accordance with one or more applicablestandards in any version.

In various implementations, display 1120 may include any flat panelmonitor or display. Display 1120 may include, for example, a computerdisplay screen, touch screen display, video monitor, television-likedevice, and/or a television. Display 1120 may be digital and/or analog.In various implementations, display 1120 may be a holographic display.Also, display 1120 may be a transparent surface that may receive avisual projection. Such projections may convey various forms ofinformation, images, and/or objects. For example, such projections maybe a visual overlay for a mobile augmented reality (MAR) application.Under the control of one or more software applications 1116, platform1102 may display user interface 1122 on display 1120.

In various implementations, content services device(s) 1130 may behosted by any national, international and/or independent service andthus accessible to platform 1102 via the Internet, for example. Contentservices device(s) 1130 may be coupled to platform 1102 and/or todisplay 1120. Platform 1102 and/or content services device(s) 1130 maybe coupled to a network 1160 to communicate (e.g., send and/or receive)media information to and from network 1160, Content delivery device(s)1140 also may be coupled to platform 1102 and/or to display 1120.

In various implementations, content services device(s) 1130 may includea cable television box, personal computer, network, telephone, Internetenabled devices or appliance capable of delivering digital informationand/or content, and any other similar device capable ofuni-directionally or bi-directionally communicating content betweencontent providers and platform 1102 and/display 1120, via network 1160or directly. It will be appreciated that the content may be communicateduni-directionally and/or bi-directionally to and from any one of thecomponents in system 1100 and a content provider via network 1160.Examples of content may include any media information including, forexample, video, music, medical and gaming information, and so forth.

Content services device(s) 1130 may receive content such as cabletelevision programming including media information, digital information,and/or other content. Examples of content providers may include anycable or satellite television or radio or Internet content providers.The provided examples are not meant to limit implementations inaccordance with the present disclosure in any way.

In various implementations, platform 1102 may receive control signalsfrom navigation controller 1150 having one or more navigation features.The navigation features of navigation controller 1150 may be used tointeract with user interface 1122, for example. In various embodiments,navigation controller 1150 may be a pointing device that may be acomputer hardware component (specifically, a human interface device)that allows a user to input spatial (e.g., continuous andmulti-dimensional) data into a computer. Many systems such as graphicaluser interfaces (GUI), and televisions and monitors allow the user tocontrol and provide data to the computer or television using physicalgestures.

Movements of the navigation features of navigation controller 1150 maybe replicated on a display (e.g., display 1120) by movements of apointer, cursor, focus ring, or other visual indicators displayed on thedisplay. For example, under the control of software applications 1116,the navigation features located on navigation controller 1150 may bemapped to virtual navigation features displayed on user interface 1122,for example. In various embodiments, navigation controller 1150 may notbe a separate component but may be integrated into platform 1102 and/ordisplay 1120. The present disclosure, however, is not limited to theelements or in the context shown or described herein.

In various implementations, drivers (not shown) may include technologyto enable users to instantly turn on and off platform 1102 like atelevision with the touch of a button after initial boot-up, whenenabled, for example. Program logic may allow platform 1102 to streamcontent to media adaptors or other content services device(s) 1130 orcontent delivery device(s) 1140 even when the platform is turned “off”In addition, chipset 1105 may include hardware and/or software supportfor 5.1 surround sound audio and/or high definition 10.1 surround soundaudio, for example. Drivers may include a graphics driver for integratedgraphics platforms. In various embodiments, the graphics driver maycomprise a peripheral component interconnect (PCI) Express graphicscard.

In various implementations, any one or more of the components shown insystem 1100 may be integrated. For example, platform 1102 and contentservices device(s) 1130 may be integrated, or platform 1102 and contentdelivery device(s) 1140 may be integrated, or platform 1102, contentservices device(s) 1130, and content delivery device(s) 1140 may beintegrated, for example. In various embodiments, platform 1102 anddisplay 1120 may be an integrated unit. Display 1120 and content servicedevice(s) 1130 may be integrated, or display 1120 and content deliverydevice(s) 1140 may be integrated, for example. These examples are notmeant to limit the present disclosure.

In various embodiments, system 1100 may be implemented as a wirelesssystem, a wired system, or a combination of both. When implemented as awireless system, system 1100 may include components and interfacessuitable for communicating over a wireless shared media, such as one ormore antennas, transmitters, receivers, transceivers, amplifiers,filters, control logic, and so forth. An example of wireless sharedmedia may include portions of a wireless spectrum, such as the RFspectrum and so forth. When implemented as a wired system, system 1100may include components and interfaces suitable for communicating overwired communications media, such as input/output (PO) adapters, physicalconnectors to connect the 110 adapter with a corresponding wiredcommunications medium, a network interface card (NIC), disc controller,video controller, audio controller, and the like. Examples of wiredcommunications media may include a wire, cable, metal leads, printedcircuit board (PCB), backplane, switch fabric, semiconductor material,twisted-pair wire, co-axial cable, fiber optics, and so forth.

Platform 1102 may establish one or more logical or physical channels tocommunicate information. The information may include media informationand control information. Media information may refer to any datarepresenting content meant for a user. Examples of content may include,for example, data from a voice conversation, videoconference, streamingvideo, electronic mail (“email”) message, voice mail message,alphanumeric symbols, graphics, image, video, text and so forth. Datafrom a voice conversation may be, for example, speech information,silence periods, background noise, comfort noise, tones and so forth.Control information may refer to any data representing commands,instructions or control words meant for an automated system. Forexample, control information may be used to route media informationthrough a system, or instruct a node to process the media information ina predetermined manner. The embodiments, however, are not limited to theelements or in the context shown or described in FIG. 11 .

As described above, system 1100 may be embodied in varying physicalstyles or form factors. FIG. 12 illustrates an example small form factordevice 1200, arranged in accordance with at least some implementationsof the present disclosure. In some examples, system 1100 may beimplemented via device 1200. In other examples, other systems,components, or modules discussed herein or portions thereof may beimplemented via device 1200. In various embodiments, for example, device1200 may be implemented as a mobile computing device a having wirelesscapabilities. A mobile computing device may refer to any device having aprocessing system and a mobile power source or supply, such as one ormore batteries, for example.

Examples of a mobile computing device may include a personal computer(PC), laptop computer, ultra-laptop computer, tablet, touch pad,portable computer, handheld computer, palmtop computer, personal digitalassistant (PDA), cellular telephone, combination cellular telephone/PDA,smart device (e.g., smartphone, smart tablet or smart mobiletelevision), mobile internet device (MID), messaging device, datacommunication device, cameras (e.g. point-and-shoot cameras, super-zoomcameras, digital single-lens reflex (DSLR) cameras), and so forth.

Examples of a mobile computing device also may include computers thatare arranged to be implemented by a motor vehicle or robot, or worn by aperson, such as a wrist computers, finger computers, ring computers,eyeglass computers, belt-clip computers, arm-band computers, shoecomputers, clothing computers, and other wearable computers. In variousembodiments, for example, a mobile computing device may be implementedas a smartphone capable of executing computer applications, as well asvoice communications and/or data communications. Although someembodiments may be described with a mobile computing device implementedas a smartphone by way of example, it may be appreciated that otherembodiments may be implemented using other wireless mobile computingdevices as well. The embodiments are not limited in this context.

As shown in FIG. 12 , device 1200 may include a housing with a front1201 and a back 1202. Device 1200 includes a display 1204, aninput/output (I/O) device 1206, a color camera 1222, and an integratedantenna 1208. For example, color camera 1222 may attain video frames forprocessing as discussed herein. Device 1200 also may include navigationfeatures 1212. I/O device 1206 may include any suitable I/O device forentering information into a mobile computing device. Examples for I/Odevice 1206 may include an alphanumeric keyboard, a numeric keypad, atouch pad, input keys, buttons, switches, microphones, speakers, voicerecognition device and software, and so forth. Information also may beentered into device 1200 by way of microphone (not shown), or may bedigitized by a voice recognition device. As shown, device 1200 mayinclude color camera 1222 and a flash 1210 integrated into back 1202 (orelsewhere) of device 1200. In other examples, color camera 1222 andflash 1210 may be integrated into front 1201 of device 1200 or bothfront and back sets of cameras may be provided.

Color camera 1222 and flash 1210 may be components of a camera module tooriginate color image data with IR texture correction that may beprocessed into an image or streaming video that is output to display1204 and/or communicated remotely from device 1200 via antenna 1208 forexample.

Various embodiments may be implemented using hardware elements, softwareelements, or a combination of both. Examples of hardware elements mayinclude processors, microprocessors, circuits, circuit elements (e.g.,transistors, resistors, capacitors, inductors, and so forth), integratedcircuits, application specific integrated circuits (ASIC), programmablelogic devices (PLD), digital signal processors (DSP), field programmablegate array (FPGA), logic gates, registers, semiconductor device, chips,microchips, chip sets, and so forth. Examples of software may includesoftware components, programs, applications, computer programs,application programs, system programs, machine programs, operatingsystem software, middleware, firmware, software modules, routines,subroutines, functions, methods, procedures, software interfaces,application program interfaces (API), instruction sets, computing code,computer code, code segments, computer code segments, words, values,symbols, or any combination thereof. Determining whether an embodimentis implemented using hardware elements and/or software elements may varyin accordance with any number of factors, such as desired computationalrate, power levels, heat tolerances, processing cycle budget, input datarates, output data rates, memory resources, data bus speeds and otherdesign or performance constraints.

One or more aspects of at least one embodiment may be implemented byrepresentative instructions stored on a machine-readable medium whichrepresents various logic within the processor, which when read by amachine causes the machine to fabricate logic to perform the techniquesdescribed herein. Such representations, known as IP cores may be storedon a tangible, machine readable medium and supplied to various customersor manufacturing facilities to load into the fabrication machines thatactually make the logic or processor.

While certain features set forth herein have been described withreference to various implementations, this description is not intendedto be construed in a limiting sense. Hence, various modifications of theimplementations described herein, as well as other implementations,which are apparent to persons skilled in the art to which the presentdisclosure pertains are deemed to lie within the spirit and scope of thepresent disclosure.

It will be recognized that the embodiments are not limited to theembodiments so described, but can be practiced with modification andalteration without departing from the scope of the appended claims. Forexample, the above embodiments may include specific combination offeatures. However, the above embodiments are not limited in this regardand, in various implementations, the above embodiments may include theundertaking only a subset of such features, undertaking a differentorder of such features, undertaking a different combination of suchfeatures, and/or undertaking additional features than those featuresexplicitly listed. The scope of the embodiments should, therefore, bedetermined with reference to the appended claims, along with the fullscope of equivalents to which such claims are entitled.

What is claimed is:
 1. A system for performing object detection,comprising: a memory to store at least a portion of one or more of acurrent, a previous, and a subsequent frame of video; and a processorcoupled to the memory, the processor to: generate a current frameinitial feature map comprising object detection scoring for the currentframe; detect, based on patch similarity, first and second pairedpatches between the current and previous frames, respectively, and thirdand fourth paired patches between the current and subsequent frames,respectively; increase a first prediction result in the current frameinitial feature map for the first paired patch to a score of the secondpaired patch or increase a second prediction result in the current frameinitial feature map for the third paired patch to a score of the fourthpaired patch to generate an enhanced feature map; and determine one ormore of an object detection localization, class, and confidence scoringfor the current frame using the enhanced feature map.
 2. The system ofclaim 1, wherein the score of the second paired patch comprises amaximum cached confidence score of the second paired patch.
 3. Thesystem of claim 2, the processor to: generate a plurality of enhancedfeature maps comprising forward object detection localization andconfidence scoring for each of a plurality of frames of the videoprevious to the previous frame and for the previous frame, wherein themaximum cached confidence score is in the enhanced feature map for theprevious frame.
 4. The system of claim 1, wherein the first predictionresult is increased to the score of the second paired patch to generatethe enhanced feature map and the second prediction result is increasedto the score of the fourth paired patch to generate a second enhancedfeature map.
 5. The system of claim 4, wherein the processor todetermine the one or more of the object detection localization, class,and confidence comprises the processor to: concatenate the enhancedfeature map and the second enhanced feature map; and provide theconcatenated feature maps to at least one neural network layer togenerate the one or more of the object detection localization, class,and confidence.
 6. The system of claim 4, wherein the processor todetermine the one or more of the object detection localization, class,and confidence comprises the processor to: determine a shared objectpatch between the enhanced feature map and the second enhanced featuremap; retain object detection localization and confidence scoring fromthe enhanced feature map for the shared object patch and discard objectdetection localization and confidence scoring from the second enhancedfeature map for the shared object patch in response to the objectdetection confidence scoring for the shared object patch in the enhancedfeature map comparing favorably to the object detection confidencescoring for the shared object patch in the second enhanced feature map;and determine the one or more of the object detection localization,class, and confidence scoring based at least on the retained objectdetection localization and confidence scoring from the enhanced featuremap.
 7. The system of claim 4, wherein the processor to determine theone or more of the object detection localization, class, and confidencecomprises the processor to: retain object detection localization andconfidence scoring for shared object patches between the enhancedfeature map and the second enhanced feature map based on higherconfidence scoring for each of the shared object patches to generate abidirectional enhanced feature map for the current frame; and determinethe one or more of the object detection localization, class, andconfidence scoring based on the bidirectional enhanced feature map.
 8. Acomputer-implemented method for performing object detection, comprising:receiving a current, a previous, and a subsequent frame of video;generating a current frame initial feature map comprising objectdetection scoring for the current frame; detecting, based on patchsimilarity, first and second paired patches between the current andprevious frames, respectively, and third and fourth paired patchesbetween the current and subsequent frames, respectively; increasing afirst prediction result in the current frame initial feature map for thefirst paired patch to a score of the second paired patch or increasing asecond prediction result in the current frame initial feature map forthe third paired patch to a score of the fourth paired patch to generatean enhanced feature map; and determining one or more of an objectdetection localization, class, and confidence scoring for the currentframe using the enhanced feature map.
 9. The method of claim 8, whereinthe score of the second paired patch comprises a maximum cachedconfidence score of the second paired patch.
 10. The method of claim 9,further comprising: generating a plurality of enhanced feature mapscomprising forward object detection localization and confidence scoringfor each of a plurality of frames of the video previous to the previousframe and for the previous frame, wherein the maximum cached confidencescore is in the enhanced feature map for the previous frame.
 11. Themethod of claim 8, wherein the first prediction result is increased tothe score of the second paired patch to generate the enhanced featuremap and the second prediction result is increased to the score of thefourth paired patch to generate a second enhanced feature map.
 12. Themethod of claim 11, wherein determining the one or more of the objectdetection localization, class, and confidence comprises: concatenatingthe enhanced feature map and the second enhanced feature map; andproviding the concatenated feature maps to at least one neural networklayer to generate the one or more of the object detection localization,class, and confidence.
 13. The method of claim 11, wherein determiningthe one or more of the object detection localization, class, andconfidence comprises: determining a shared object patch between theenhanced feature map and the second enhanced feature map; retainingobject detection localization and confidence scoring from the enhancedfeature map for the shared object patch and discard object detectionlocalization and confidence scoring from the second enhanced feature mapfor the shared object patch in response to the object detectionconfidence scoring for the shared object patch in the enhanced featuremap comparing favorably to the object detection confidence scoring forthe shared object patch in the second enhanced feature map; anddetermining the one or more of the object detection localization, class,and confidence scoring based at least on the retained object detectionlocalization and confidence scoring from the enhanced feature map. 14.The method of claim 11, wherein determining the one or more of theobject detection localization, class, and confidence comprises:retaining object detection localization and confidence scoring forshared object patches between the enhancement feature map and the secondenhanced feature map based on higher confidence scoring for each of theshared object patches to generate a bidirectional enhanced feature mapfor the current frame; and determining the one or more of the objectdetection localization, class, and confidence scoring based on thebidirectional enhanced feature map.
 15. At least one non-transitorymachine readable medium comprising a plurality of instructions that, inresponse to being executed on a computing device, cause the computingdevice to perform object detection by: receiving a current, a previous,and a subsequent frame of video; generating a current frame initialfeature map comprising object detection scoring for the current frame;detecting, based on patch similarity, first and second paired patchesbetween the current and previous frames, respectively, and third andfourth paired patches between the current and subsequent frames,respectively; increasing a first prediction result in the current frameinitial feature map for the first paired patch to a score of the secondpaired patch or increasing a second prediction result in the currentframe initial feature map for the third paired patch to a score of thefourth paired to generate an enhanced feature map; and determining oneor more of an object detection localization, class, and confidencescoring for the current frame using the enhanced feature map.
 16. Thenon-transitory machine readable medium of claim 15, wherein the score ofthe second paired patch comprises a maximum cached confidence score ofthe second paired patch.
 17. The non-transitory machine readable mediumof claim 16, further comprising: generating a plurality of enhancedfeature maps comprising forward object detection localization andconfidence scoring for each of a plurality of frames of the videoprevious to the previous frame and for the previous frame, wherein themaximum cached confidence score is in the enhanced feature map for theprevious frame.
 18. The non-transitory machine readable medium of claim15, wherein the first prediction result is increased to the score of thesecond paired patch to generate the enhanced feature map and the secondprediction result is increased to the score of the fourth paired patchto generate a second enhanced feature map.
 19. The non-transitorymachine readable medium of claim 18, wherein determining the one or moreof the object detection localization, class, and confidence comprises:concatenating the enhanced feature map and the second enhanced featuremap; and providing the concatenated feature maps to at least one neuralnetwork layer to generate the one or more of the object detectionlocalization, class, and confidence.
 20. The non-transitory machinereadable medium of claim 18, wherein determining the one or more of theobject detection localization, class, and confidence comprises:determining a shared object patch between the enhanced feature map andthe second enhanced feature map; retaining object detection localizationand confidence scoring from the enhanced feature map for the sharedobject patch and discard object detection localization and confidencescoring from the second enhanced feature map for the shared object patchin response to the object detection confidence scoring for the sharedobject patch in the enhanced feature map comparing favorably to theobject detection confidence scoring for the shared object patch in thesecond enhanced feature map; and determining the one or more of theobject detection localization, class, and confidence scoring based atleast on the retained object detection localization and confidencescoring from the enhanced feature map.
 21. The non-transitory machinereadable medium of claim 18, wherein determining the one or more of theobject detection localization, class, and confidence comprises:retaining object detection localization and confidence scoring forshared object patches between the enhanced feature map and the secondenhanced feature map based on higher confidence scoring for each of theshared object patches to generate a bidirectional enhanced feature mapfor the current frame; and determining the one or more of the objectdetection localization, class, and confidence scoring based on thebidirectional enhanced feature map.